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FOREWORD 


The  Fort  Leavenworth  Field  Unit  of  the  U.S.  Army  Research  Institute 
conducts  a  systems  and  training  research  program  in  support  of  the  Combined 
Arms  Center.  From  1976  through  1984  the  Field  Unit  worked  on  determining 
training  objectives,  developing  performance  measures,  evaluating  training 
system  effectiveness,  and  analyzing  the  effects  of  feedback  on  command  group 
performance.  This  research  was  performed  in  cooperation  with  the  military 
developers  and  operators  of  autom  ted  battle  simulation  systems  used  to  train 
battalion  command  groups  in  the  command  and  control  of  combined  arms  opera¬ 
tions,  especially  CATTS,  the  Combined  Arms  Tactical  Training  Simulator,  CAMMS, 
the  Computer  Assisted  Map  Maneuver  System,  and  ARTBASS,  the  Army  Training  Bat¬ 
tle  Simulation  System. 

The  reporting  of  these  lessons  learned  was  conducted  as  part  of  the  ARI 
Research  Task  1.3.3.,  Improved  Methods  for  Command  Group  Training.  The  work 
was  performed  under  agreement  with  the  Command  and  General  Staff  College, 
Memorandum  of  Understanding  entitled  "Research  and  Evaluation  Program  for 
Present  and  Future  Command  and  Control  Requirements  and  Operations,"  dated  31 
May  1983.  The  results  of  this  research  were  briefed  to  the  Director,  Training 
Support  Directorate  of  the  Combined  Arms  Training  Activity  on  1  August  1985. 
The  lessons  learned  in  this  research  are  being  applied  to  the  development  of 
command  group  training  systems  at  the  corps  and  division  levels. 
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LESSONS  LEARNED  IN  RESEARCH  ON  COMMAND  GROUP  TRAINING 


EXECUTIVE  SUMMARY 


Objective: 

The  purpose  of  this  report  is  to  summarize  lessons  learned  during  several 
years  of  research  on  the  development  of  automated  battle  simulations  for 
training  battalion  command  groups. 


Procedures: 

Data  vere  collected  at  over  100  command  group  training  exercises.  Mea¬ 
sures  of  command  group  performance  and  training  system  effectiveness  vere 
obtained  by  means  of  surveys,  interviews,  multiple-choice  questionnaires, 
task- performance  ratings,  responses  to  critical  events  (probes),  and  simula¬ 
tion  outcome  measures. 


Findings: 

The  lessons  learned  in  this  research  vere  grouped  under  four  headings: 
training  objectives,  performance  measures,  training  effectiveness,  and 
feedback. 

The  objectives  of  a  training  system  should  be  determined  empirically, 
i.e.,  by  observing  and  interviewing  the  participants  at  training  exercises.  A 
battle  simulation  does  not  necessarily  train,  or  need  to  train,  all  the  tasks 
that  a  command  group  performs  in  combat.  A  training  system’s  objectives  may 
be  defined  as  the  subset  of  tasks  that  are  both  strongly  related  to  overal?. 
effectiveness  and  commonly  deficient.  Training  objectives,  so  defined,  vary 
with  the  unit’s  mission.  They  also  depend  on  the  command  group  members’  ex¬ 
perience  and  goals.  The  system’s  and  the  command  group’s  common  objectives 
should  determine  the  content  of  the  training  exercise  and  the  feedback  given 
to  the  members. 

A  variety  of  measures  should  be  used  to  assess  different  aspects  of 
command  group  performance.  Performance  ratings  are  easy  to  obtain  from 
exercise  controllers  and  from  the  players  themselves,  once  suitable  rating 
scales  have  been  developed.  But  their  usefulness  is  limited  by  the  rater’s 
ability  to  discriminate  among  different  tasks,  as  veil  as  by  individual  dif¬ 
ferences  among  raters.  More  objective  measures  have  been  developed  to  over¬ 
come  these  limitations,  Including  measures  of  information  flow,  responses  to 
critical  events  (probes),  and  simulation  outcome  measures.  These  measures  are 
useful  for  providing  players  with  feedback  to  help  them  Improve  their  perfor¬ 
mance  and  for  providing  the  trainers  with  feedback  to  help  improve  the  train¬ 
ing  system.  Further  research  should  be  done  to  develop  objective  measures  of 
planning  and  decision-making  behavior,  to  develop  automated  measures  of 
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performance*  and  to  make  use  of  advances  In  audio*  and  video-recording 
technology. 

Automated  battle  simulations  create  a  realistic  experience,  but  they 
train  some  command  group  members  more  than  others.  Commanders,  operations, 
fire  support,  and  intelligence  are  trained  better  than  logistics  and  admin* 
istration.  It  vould  be  desirable  to  develop  modular  systems  that  train  in¬ 
dividual  staff  sections.  In  addition  to  providing  each  staff  element  with 
optimally  focused  training,  such  part-staff  trainers  could  be  used  to  provide 
supplemental  training  to  less  experienced  staff  members.  At  corps  and  divi¬ 
sion  levels,  part-staff  traineis  vould  ba  an  economical  alternative  to  large- 
scale  exercises  for  the  entire  staff. 

Merely  participating  in  an  exercise  does  not  necessarily  improve  command 
group  performance.  To  maximire  the  training  benefit  of  an  exercise,  the  group 
members  should  be  given  extensive  feedback  about  their  performance.  In  par¬ 
ticular,  they  should  be  given  feedback  relevant  to  their  training  objectives. 
The  optimum  content,  duration,  and  format  of  feedback  remains  to  be  deter¬ 
mined,  but  it  should  incorporate  the  playback  capability  of  automated  simu¬ 
lations,  audio-video  inputs,  and  objective  performance  measures,  as  they 
continue  to  be  developed. 


Utilization  of  Findings: 

The  lessons  learned  in  this  research  vere  used  in  the  development  and 
operation  of  battalion  command  group  training  systems.  They  can  also  be  used 
to  facilitate  the  development  of  future  systems  to  train  command  groups  at 
corps  8nd  division  levels. 
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LESSONS  LEARNED  IN  RESEARCH  ON  COMMAND  GROUP  TRAINING 


INTRODUCTION 

This  paper  summarises  thv  lessons  learned  in  research  on  the  development 
of  command  group  training  systems,  conducted  by  the  Army  Research  Institute  at 
Fort  Leavonvorth  from  1976  to  198*  The  major  systems  studied  during  this 
time  vere  CATTS,  the  Combined  Arms  Tactical  Training  Simulator;  CAMMS,  the 
Computer  Assisted  Map  Maneuver  System;  and  ARTBASS,  the  Army  Training  battle 
Simulation  System.  Since  the  methods  and  results  summarised  in  this  paper 
came  from  research  on  training  systems,  they  can  readily  be  grouped  into  cate¬ 
gories  derived  from  a  model  of  training  system  development.  In  addition  to 
providing  a  framevork  for  organising  the  results,  the  model  also  Indicates 
where  the  methods  can  be  applied  in  the  development  of  future  command  group 
training  systems.  Therefore  the  lessons  learned  will  be  presented  in  the 
context  of  a  model  of  training  system  development,  which  is  diagrammed  in 
Figure  1. 


Figure  1.  A  model  of  training  system  development 


The  first  step  in  the  model  is  to  determine  the  objectives  of  the  train¬ 
ing  system,  i.e.,  the  tasks  that  need  to  be  trained.  These  objectives  support 
the  design  of  the  training  system  and  the  development  of  performance  measures, 
which  are  the  next  two  steps  in  the  model.  The  measures  in  turn  are  used  to 
assess  performance  before,  during,  and  after  training.  Then  the  results  are 
used  to  provide  feedback  to  help  the  trainees  improve  their  performance,  to 
help  the  designers  increase  the  effectiveness  of  the  training  system,  and  to 
refine  the  original  training  objectives.  It  is  important  to  note  that  this 
model  for  training  system  development  applies  also  to  training  system  utili¬ 
zation.  Given  an  existing  training  system,  for  example,  it  is  typically  ad¬ 
visable  to  "design”  a  system  configuration  (to  Include  type  of  scenario, 
length  of  the  exercise,  number  of  trainees,  etc.),  appropriate  to  the  train¬ 
ing  objective  of  a  given  group  of  trainees.  The  lessons  learned  in  the  ARI 
research  program  apply  to  both  training  system  development  and  utilization, 
and  the  model  in  Figure  1  should  be  integrated  in  this  broader  context.  In 
the  framevork  oi  this  model,  principles  of  command  group  training  vill  be 
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presented  under  the  headings  of  lasaons  about  (1)  training  objectives,  (2) 
parforaanca  measurement,  (3)  training  effectiveness,  and  (A)  feedback. 


TRAINING  OBJECTIVES 

Thara  ara  aavaral  raasona  vhy  it  is  important  to  opacity  tha  objactlvas 
of  a  training  system:  (1)  to  dafina  tha  purposa  of  tha  system,  (2)  to  davalop 
parforaanca  measures,  (3)  to  conduct  axarcisas  that  maat  tha  objectives  of  tha 
individuals  vho  ara  trained,  (A)  to  provide  faadback  that  addrassas  those  ob¬ 
jectives,  (5)  to  evaluate  the  system's  effectiveness,  and  (6)  to  provide  tha 
system  developers  vith  faadback  about  hov  vail  the  system  satisfies  its  ob¬ 
jactlvas.  It  is  essential  that  tha  system  developers,  the  controllers  vho 
conduct  tha  axarcisas,  and  tha  command  groups  that  participate  in  them  all 
share  a  common  understanding  of  tha  training  objectives. 

Tha  general  purposa  of  a  command  grotto  training  system  is  to  train  tha 
commander  and  his  staff  in  tha  control  and  coordination  of  combined,  arms 
operations.  But  this  statement  is  not  specific  enough  to  support  the  design 
of  n  training  system  or  tha  development  of  performance  measures.  A  much  more 
detailed  description  of  potential  training  objectives  is  given  in  the  Army 
Training  and  Evaluation  Program.  ARTEP  71-2  contains  a  list  of  training  ob¬ 
jectives  (tasks,  conditions,  and  standards)  that  a  battalion  command  group 
must  achieve  to  perform  successfully  on  tha  battlefield.  The  1977  edition  of 
ARTEP  71-2  vas  used  in  an  investigation  conducted  to  dafina  tha  training  ob¬ 
jectives  of  CATTS  (Barber  &  Kaplan,  1979).  Taking  the  command  group  module  of 
ARTEP  71-2  as  a  starting  point,  this  investigation  sought  to  discover  which 
tasks  vere  exercised  in  CATTS,  vhlch  tasks  vere  most  important,  in  the  sense 
that  the>  vere  strongly  related  to  overall  performance,  and  vhlch  tasks  most 
needed  to  be  trained,  because  they  received  relatively  lov  performance  rat¬ 
ings.  The  CATTS  controllers  used  tha  ARTEP  tasks  and  standards  to  rate  the 
performance  of  27  battalion  command  groups  vith  the  results  described  under 
Lessons  1,  2,  3,  and  A. 

LESSON  1  -  Simulations  did  not  train  all  the  tasks  the  command  group 
needed  to  perform.  One  of  the  first  lessons  learned  In  the  attempt  to  measure 
task  performance  vas  that  CATTS  did  not  exercise  all  the  ARTEP  tasks.  CATTS 
did  train  most  of  the  tasks  that  a  battalion  commander  and  his  staff  vould 
perform  in  their  command  post.  Eighteen  percent  of  the  tasks  (11  of  61 
tasks),  hovever,  vere  not  exercised  in  CATTS.  Some  tasks  vere  not  exercised 
because  they  could  only  be  performed  in  the  field,  e.g.,  camouflage.  Other 
tasks  vere  not  modeled  in  the  simulation,  e.g.,  react  to  chemical,  biological, 
or  nuclear  attack. 

LESSON  2  -  Certain  tasks  vere  particularly,  important  for  command  group 
effectiveness.  Fifty  tasks  vere  too  many  to  observe,  evaluate,  and  provide 
feedback  about.  Therefore,  it  vas  necessary  to  identify  a  smaller  set  of 
tasks  as  training  objectives.  One  criterion  used  to  Identify  training  ob¬ 
jectives  vas  that  of  being  significantly  correlated  vith  ratings  of  overall 
effectiveness.  About  half  the  tasks  exercised  in  CATTS  vere  found  to  be  im¬ 
portant  by  that  criterion.  Ratings  for  23  of  the  50  tasks  exercised  vere 
significantly  correlated  (p  <  .01)  vith  overall  effectiverc^  ratings,  i.e., 
these  tasks  vere  strongly  associated  vith  overall  effectiveness.  Most  of 
these  tasks  vere  performed  by  the  battalion  commander,  the  S2,  and  the  S3. 
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*  Certain  tasks  received  low  performance  ratings.  A  second 


criterion  used  to  identify  training  objectives  was  that  they  were  frequently 
deficient.  Nineteen  tasks  were  identified  as  deficient  because  they  received 
mean  ratings  which  were  more  than  one  standard  deviation  below  the  mean  of  all 
50  tasks  that  were  exercised  in  CATTS.  Figure  2  illustrates  the  relationships 
among  several  subsets  of  tasks:  those  which  were  exercised,  important,  and 
deficient.  Of  particular  inter  st  is  the  subset  of  14  t*«:ks  which  were  both 
important  and  deficient. 


Figure  2.  Classification  of  ARTEP  tasks 


LESSON  4  -  Tasks  that  were  both  important  and  deficient  were  classified 
as  critical  training  objectives.  A  second  investigation  of  23  battalion 
command  groups  at  CATTS  exercises  (Kaplan  &  Barber,  1979)  yielded  results 
similar  to  those  described  above.  Together  the  two  investigations  identified 
a  total  of  20  ARTEP  tasks  that  were  significantly  correlated  with  overall 
performance  ana  ’■hat  also  received  relatively  low  performance  ratings.  These 
tasks  were  identified  as  critical  training  objectives  for  battalion  command 
groups  (see  Table  1). 
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Table  1 


Tasks  Identified  as  Critical  Battalion  Command  Group  Training  Objectives 


Barber  & 

Kaplan  & 

Kaplan, 

Barber, 

Task 

1979 

1979 

Task  1. 

Develop  plan  based  on  mission 

1-B. 

Identify  critical  combat  information 

and  intelligence 

X 

X 

1-1. 

Plan  fires 

X 

X 

Task  2. 

Initiate  intelligence  preparation  of  the 

battlefield 

2-A. 

Identify  critical  combat  information  and 

intelligence 

X 

X 

2-B. 

Gather  critical  combat  information  and 

intelligence 

X 

X 

2-C. 

Analyze  opposing  force 

X 

2-D. 

Disseminate  critical  combat  information 

and  intelligence 

X 

Task  3. 

Prepare  and  organize  the  battlefield 

3-G. 

Communicate/coordinate  plans  and  orders 

X 

X 

Task  5. 

See  the  battlefield  during  the  battle 

S-B. 

Gather  critical  combat  information  and 

intelligence 

X 

5-C. 

Analyze  opposing  force 

X 

5-D. 

Disseminate  critical  combat  information 

and  intelligence 

X 

Task  6. 

Control  and  coordinate  combat  operations 

6-B. 

Coordinate/communicate  changes 

X 

Task  7. 

Employ  fires  and  other  combat  support  assets 

7-A. 

Modify  fire  support  plan 

X 

Task  8. 

Concentrate/shift  combat  power 

8-A. 

Determine  critical  place  and  time 

X 

X 

8-B. 

Concentrate/shift  combat  power  in  the  attack 

X 

8-C. 

Concentrate/shift  combat  power  in  the 

defense  or  retrograde 

X 

X 

Task  9. 

Manage  combat  service  support  assets 

9-A. 

Arm  and  fuel  the  systems 

X 

9-D. 

Integrate  CSS  into  scheme  of  maneuver 

X 

Task  10. 

Secure  and  protect  the  TF 

10-A. 

Defeat  or  suppress  opposing  force’s  electro- 

magnetic  intelligence  effort 

X 

X 

Task  11. 

Troop  lead  during  battle 

1 1  -A. 

Supervise  compliance  with  TF  order 

X 

Task  12. 

React  to  situations  requiring  special 

actions 

12-A. 

React  to  opposing  force  electronic  warfare 

X 

X 
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LESSON  5  -  The  tasks  Identified  as  training  objectives  varied  vith  thg 
mission.  Some  tasks  were  more  difficult  to  perform  In  one  mission  than  in 
another.  Kaplan  and  Barber  (1979)  observed  that  more  tasks  received  low 
performance  ratings  during  a  covering  force  operation  than  during  an  attack  or 
a  defense.  Consequently,  more  tasks  vere  identified  as  critical  for  training 
in  the  covering- force  operation  than  in  the  other  missions.  All  but  one  (6-B) 
of  the  tasks  in  Table  1  vere  critical  in  the  covering  force  operation,  five 
(2-B,  2-D,  3-G,  6-B,  10-A)  vere  critical  in  a  mechanized  attack,  one  (2-B)  in 
a  nonmechanized  infantry  attack,  one  (9-A)  in  the  defense,  and  one  (2-B)  in 
the  nonmechanized  infantry  attack  mission. 

LESSON  6  -  Each  command  group  had  its  own  training  objectives.  In  a 
later  investigation  members  of  12  battalion  command  groups  selected  their  ovn 
training  objectives  from  a  list  of  ARTEP  tasks  (ARTEP  71-2,  1981)  before  par¬ 
ticipating  in  ARTBASS  exercises  (Kaplan,  1984).  The  number  of  objectives 
chosen  reflected  the  participant’s  level  of  training.  An  experienced  com¬ 
mander,  vhose  battalion  had  just  completed  its  ARTEP,  chose  only  1  of  the  20 
objectives  available  to  him,  although  he  added  8  of  his  ova  to  the  list. 

Other  commanders  chose  all  20  objectives.  A  nev  S4,  vho  chose  all  six  objec¬ 
tives  available  to  him,  commented:  "I  have  been  in  the  unit  for  2  veeks. 
Therefore,  all  areas  require  vork  and  familiarization  on  my  part." 

Some  objectives  vere  chosen  by  almost  everyone.  Over  902  of  the  bat¬ 
talion  commanders  vanted  training  to  Conduct  commander  and  staff  planning 
procedures  and  to  Control  and  coordinate  Bn/TF  operations.  A  similar  per¬ 
centage  of  S2s  selected  Obtain  information  and  intelligence.  Most  S3s  chose 
Prepare  plan/order.  S4s  chose  Plan  and  coordinate  logistical  support,  and  FSOs 
vanted  to  Develop  fire  support  plans.  There  vas  less  unanimity  among  the  Sis: 
the  most  popular  objectives,  Process  replacements  and  Prepare  and  submit  per¬ 
sonnel  daily  summary  (PDS1.  vere  chosen  by  only  642  of  the  Sis. 

The  training  objectives  selected  for  each  position  are  listed  in  order  of 
decreasing  frequency  in  Appendix  A.  It  is  interesting  to  note  that  many  of 
the  most  frequently  chosen  tasks  vere  performed  during  the  planning  phase 
before  the  simulated  battle  began,  l.e.,  they  could  be  trained  vithout  the 
simulator. 

Recommends t ions.  The  objectives  of  a  training  system  should  be  deter¬ 
mined  empirically,  i.e.,  by  observing  and  lntervleving  the  participants  at 
training  exercises.  A  battle  simulation  does  not  necessarily  train,  or  need 
to  train,  all  the  tasks  that  a  command  group  performs  in  combat.  A  training 
system’s  objective  may  be  defined  as  the  subset  of  tasks  that  are  both 
strongly  related  to  overall  effectiveness  and  commonly  deficient.  Training 
objectives,  so  defined,  vary  vith  the  unit’s  mission.  They  also  depend  on  the 
command  group  members’  experience  and  goals.  The  system’s  and  the  command 
group’s  common  objectives  should  determine  the  content  of  the  training  exer¬ 
cise  and  of  the  feedback  that  is  given  to  the  members. 


PERFORMANCE  MEASURES 

Four  types  of  performance  measures  vere  used  in  the  Field  Unit’s  research 
on  command  group  training:  (1)  ratings  of  ARTEP  tasks,  (2)  measures  of  infor¬ 
mation  flov,  (3)  responses  to  probes,  and  (4)  simulation  outcomes.  The 
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lessons  learned  about  rating  ARTEP  tasks  include  what  rating  scale  to  use,  hov 
reliable  performance  ratings  are,  and  how  well  ratings  discriminate  the  per¬ 
formance  of  different  tasks.  In  addition,  several  lessons  were  learned  about 
coumunication  in  command  groups. 

LESSON  7  -  Performance  rating  scales  must  be  suitably  labeled  and  have  an 
appropriate  number  of  steps.  The  ARTEP  itself  recognizes  only  two  levels  of 
performance:  satisfactory  and  unsatisfactory.  Because  the  CATTS  controllers 
were  reluctant  to  characterize  a  command  group’s  performance  as  unsatisfac¬ 
tory,  Barber  &  Kaplan  (1979)  provided  them  with  a  three-point  scale,  which  was 
defined  as  follows: 

1  -  Major  departure  from  ARTEP  standard,  unsatisfactory. 

2  -  Minor  deviation  from  ARTEP  standard. 

3  -  Satisfies  ARTEP  standard. 

After  gaining  experience  with  the  three-point  scale,  the  controllers 
asked  for  a  scale  that  would  let  them  make  finer  distinctions.  Therefore  the 
following  five-point  scale  was  introduced  to  increase  the  resolution  of  their 
task  performance  ratings  (Kaplan  &  Barber,  1979): 

1  -  Completely  overlooked,  forgotten. 

2  -  Major  deficiencies. 

3  -  Minor  deficiencies. 

4  -  Satisfactory. 

5  -  Excellent. 

Adding  steps  to  a  scale  beyond  the  rater's  ability  to  make  distinctions 
does  not  increase  the  resolution  of  the  ratings.  In  one  investigation  a  mag¬ 
nitude  estimation  procedure  was  used.  This  procedure  defined  satisfactory 
performance  as  "100."  Performance  twice  as  good  was  rated  200;  half  as  good 
was  rated  50.  The  magnitude  estimation  procedure  (Thomas,  Barber,  &  Kaplan, 
1984)  and  a  nine-point  scale  (Thomas,  Kaplan,  &  Barber  1984)  gave  as  good  but 
not  better  resolution  than  the  five-point  scale.  The  usefulness  of  a  perfor¬ 
mance-rating  procedure  is  limited  by  its  reliability  (see  Lesson  9). 

LESSON  8  -  Raters  did  not  discriminate  well  among  different  ARTEP  tasks. 
That  is  to  say,  when  command  group  performance  was  rated  high  on  one  ARTEP 
task,  it  was  usually  rated  high  on  most  other  tasks  as  well.  Factor  analysis 
of  ratings  from  controllers,  players  (the  command  group  members),  and  player- 
controllers  (company  commanders  and  fire  support  team  members)  Indicated  that 
just  a  few  types  of  performance  were  distinguished  (Thomas,  Barber,  &  Kaplan, 
1984).  Only  one  factor  was  derived  from  the  controllers’  ratings,  which 
suggests  that  they  did  not  discriminate  among  the  tasks.  Analysis  of  the 
players’  ratings  yielded  a  three-factor  solution.  Factor  I  was  equivalent  to 
Gather  and  analyze  required  information.  Factor  II  combined  Develop  a  plan 
based  on  mission  and  modify  it  as  required  bv  events  with  Communicate/ 
coordinate.  Factor  III  corresponded  to  Implement  plan  and  Supervise  combat 
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operations.  The  player-controllers’  ratings  loaded  on  two  factors:  Factor  I 
Included  Implement  plan  and  Supervise  combat  operations:  Factor  II  Included 
Gather  and  analyze  required  Information.  Communicate/coordinate,  and  tvo- 
thirds  of  the  tasks  grouped  under  Develop  a  plan  based  on  mission  and  modify 
It  as  required  bv  events. 

On  the  basis  of  the  above  results,  the  ARTEP  rating  form,  vhlch  had  once 
been  expanded  to  85  Items  (Barber  &  Kaplan,  1979),  vas  reduced  to  the  follow¬ 
ing  five  tasks  In  subsequent  research  (Thomas,  Kaplan,  &  Barber,  1984): 

1.  Gather  and  analyze  required  Information.  Includes:  (a)  analyze 
mission,  (b)  determine  what  information  is  available  and  vhat  additional  in¬ 
formation  is  required,  (c)  determine  vhat  information  sources  are  available, 
and  (d)  gather  all  available  information  and  request  additional  information  as 
needed . 


2.  Develop  a  plan  based  on  mission  and  modify  it  as  required  bv  events. 
Includes:  (a)  determine  friendly  capabilities  and  limitations,  request  addi¬ 
tional  assets  if  needed,  (b)  estimate  enemy  capabilities  and  likely  courses  of 
action,  (c)  identify  key  terrain,  (d)  select  battle  position/routes  to  objec¬ 
tives,  (e)  identify  critical  place,  (f)  develop  and  compare  courses  of  action, 
(g)  individual  staff  planning  for  communications,  intelligence,  operations, 
admin/log,  fires,  and  (h)  coordinate  with  other  staff  members. 

3.  Communicate/coordinate .  Includes:  (a)  issue  a  warning  order,  (b) 
disseminate  plans  and  orders,  and  (c)  disseminate  combat  information  and 
intelligence  to  higher  and  lover. 

4.  Implement  plan.  Includes:  (a)  concentrate/shift  combat  pover  and 
(b)  reinforce  terrain. 

5.  Supervise  combat  operations.  Includes:  (a)  compare  battlefield 
events  vlth  current  order  and  concept  of  operations,  (b)  determine  that  a  nev 
course  of  action  is  necessary,  and  (c)  determine  that  a  change  in  implementa¬ 
tion  is  necessary. 

LESSON  9  -  Task  performance  ratings  vere  not  very  reliable.  Raters  often 
disagreed  about  hov  veil  a  task  vas  performed.  The  coefficient  of  reliabil¬ 
ity,  vhlch  can  vary  from  -1  to  +1,  averaged  .22  for  a  single  rater  using  the 
above  five-point  scale.  One  response  to  this  problem  vas  to  have  several 
observers  rate  the  same  tasks.  Then  the  reliability  of  the  ratings  could  be 
calculated.  Moreover,  the  mean  of  several  ratings  Is  more  reliable  than  a 
single  rating.  In  the  present  instance,  the  coefficient  of  reliability  in¬ 
creased  from  .22  for  ratings  from  one  observer  to  .55  for  the  mean  ratings 
from  five  observers.  Another  response  to  the  problem  of  lov  rater  reliability 
vas  to  develop  more  objective  measures  of  performance:  the  information-flov, 
probe,  and  simulation-outcome  measures  described  belov. 

LESSON  10  -  Much  information  vas  lost  in  the  process  of  communication. 

The  first  indication  that  communication  vas  a  problem  came  from  ratings  of 
ARTEP  tasks.  Several  of  the  tasks  in  Table  1,  vhlch  vere  commonly  deficient 
and  strongly  related  to  overall  performance,  involved  communication:  espe¬ 
cially  Gather  critical  combat  information  and  intelligence.  Disseminate 
critical  combat  information  and  intelligence.  Communicate/coordinate  plans  and 
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orders,  and  oordlnate/communicate  changes.  Later  a  method  was  developed  to 
trace  the  flow  of  Information  through  battalion  command  groups  (Kaplan,  1980). 
It  used  a  multiple-choice  questionnaire  to  measure  the  reception  of  informa¬ 
tion  objectively.  An  example  of  an  information-flow  questionnaire  is  given  in 
Appendix  B.  Data  from  13  groups  shoved  thav  the  members  received  81X  of  the 
Information  presented  to  them  by  brigade,  but  only  48X  of  the  information  that 
they  needed  to  receive  from  other  members  of  their  own  command  group. 

LESSON  11  -  There  were  large  individual  differences  in  communication 
performance.  Some  people  were  much  better  communicators  than  others.  Some 
were  better  transmitters;  others  were  better  receivers.  Transmission  and 
reception  scores  obtained  at  CATTS  exercises  with  the  information-flow  meth¬ 
odology  ranged  from  near  zero  to  100X.  Battalion  commanders  were  generally 
effective  communicators,  but  their  score.*,  too,  varied  over  a  vide  range.  In 
a  sample  of  32  command  groups,  the  commanders'  transmission  scores  varied  from 
42  to  92X,  and  their  reception  scores  from  35  to  93X,  as  shown  in  Figure  3 
(Koontz  &  Kaplan,  1933). 
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Figure  3.  Distribution  of  communication  scores  from  32  battalion  command 
groups.  The  percent  of  required  information  received  by  the 
commander  from  his  staff  is  plotted  against  the  percent  of  re¬ 
quired  information  transmitted  by  the  commander  and  his  S3 
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LESSON  12  -  Effective  communication  vas  associated  with  certain  oh^prv. 
able  behaviors.  The  commanders*  communication  patterns  were  related  to  their 
leadership  styles.  Authoritarian  or  autocratic  commanders  had  low  transmis¬ 
sion  and  reception  scores:  They  did  not  seek  information  from  their  staffs, 
nor  did  they  explain  their  orders  to  them.  Permissive  commanders,  who  let 
their  staffs  do  most  of  the  work,  also  had  low  communication  scores.  Com¬ 
manders  who  supervised  their  staffs  closely  had  higher  transmission  scores. 
Commanders  who  managed  by  exception  had  higher  reception  scores:  They  lis¬ 
tened  to  their  staffs  attentively,  but  gave  orders  only  to  prevent  errors. 
Commanders  who  used  a  more  consultative  or  participative  style,  i.e.,  who 
interacted  with  their  staffs  during  the  decision-making  process,  had  high 
transmission  and  reception  scores. 

Figure  4  illustrates  the  relationship  between  communication  and  leader¬ 
ship  style.  The  horizontal  axis  indicates  leader  behaviors  associated  with 
low  to  high  transmission,  while  the  vertical  axis  identifies  the  leader  be¬ 
haviors  associated  with  reception.  The  straight  and  curved  arrows  on  the 
graph  represent  the  loci  of  communication  (transmission  and  reception)  scores 
on  a  continuum  of  leadership  styles  between  autocratic,  participative,  and 
permissive  leadership. 
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Figure  4.  The  relationship  between  communication  and  leadership  style 


Other  behaviors  that  vere  related  to  communication  performance  were  ac¬ 
tive  listening,  asking  questions,  and  taking  notes,  all  of  vhich  vere  asso¬ 
ciated  vith  high  reception  scores.  Careful  preparation  of  briefings  and  the 
use  of  structured  briefing  formats  contributed  to  high  transmission  scores. 
Calling  the  command  group  together  to  receive  the  commander’s  guidance,  to 
present  staff  estimates,  and  to  receive  the  commander’s  concept  facilitated 
both  transmission  and  reception. 

Providing  group  members  vith  feedback  about  their  information-flow  scores 
enabled  them  to  improve  their  communication  performance  (Thomas,  Kaplan,  & 
Barber,  1984).  The  effects  of  feedback  are  discussed  under  Lessons  18-20, 
belov. 

LESSON -13  -  .Command  group  Performance  could  be  assessed  by  the  members* 
responses  to  critical  events .  Thomas,  Kaplan,  &  Barber  (1984)  designed 
“probes'*  to  elicit  coordination,  communication,  and  information-processing 
behaviors  within  the  command  group.  They  vere  Inserted  at  appropriate  times 
in  the  exercise  by  controllers,  who  evaluated  the  timeliness,  completeness, 
and  accuracy  of  the  responses.  Figure  5  is  an  example  of  a  probe  inserted  by 
the  brigade  S3  controller  to  measure  the  performance  of  the  battalion  S3. 
Probes  vere  developed  for  the  SI,  S2,  S4,  and  FSO,  as  veil  as  for  the  S3. 

Some  probes  vere  based  on  requests  for  information  by  the  brigade  controllers; 
other  probes  came  from  subordinate  units.  Thus,  probes  for  the  SI  included  a 
report  of  the  loss  of  a  company  commander  and  of  an  incomplete  strength  re¬ 
port.  The  S2  received  a  report  of  a  suspected  chemical  attack.  The  S4  re¬ 
ceived  a  repair  order  from  a  contact  team  and  a  report  of  a  defective  fuse, 
and  the  FSO  received  a  report  of  firing  on  a  friendly  unit.  Examples  of  other 
probes  are  given  in  Appendix  C. 

LESSON  14  -  Command  group  performance  could  be  measured  bv  simulated 
battlefield  outcomes.  Thomas,  Barber,  &  Kaplan  (1984)  used  various  ways  of 
combining  friendly  and  0PF0R  strength  and  losses  to  indicate  hov  successful 
the  friendly  forces  vere  in  the  simulated  battle.  Four  such  measures  are 
defined  in  Table  2.  The  Relative  Exchange  Ratio  (RER)  and  Surviving  Maneuver 
Force  Ratio  Differential  (SMFRD)  are  variations  of  measures  used  in  CATTS 
exercises.  RER  is  simply  the  ratio  of  the  proportion  of  0PF0R  losses  to  the 
proportion  of  friendly  losses.  SMFRD  is  calculated  by  subtracting  the  pro¬ 
portion  of  0PF0R  surviving  battle  from  the  proportion  of  friendly  forces 
surviving.  Both  measures  are,  therefore,  rather  straightforward  comparisons 
of  losses  or  surviving  strengths  of  opposing  forces. 

The  Command  and  Control  Index  of  Lethality  Levels  (C^ILL)  is  based  on  the 
assumption  that  it  is  preferable  to  have  a  high  percentage  of  forces  surviv¬ 
ing,  while  attriting  a  relatively  high  proportion  of  enemy  forces.  Hence, 
C^ILL  is  computed  by  adding  the  tvo  components  together.  The  proportion  of 
friendly  forces  surviving  is  divided  in  half  because  it  vas  observed  that  in 
covering  force  missions,  controller  ratings  of  performance  vere  more  respon¬ 
sive  to  the  amount  of  enemy  forces  attrited  than  to  the  amount  of  friendly 
forces  surviving.  The  weighting  factor  places  a  higher  emphasis  on  0PF0R 
losses. 

The  Change  in  Combat  Ratio  (ACR)  is  based  on  the  assumption  that  it  is 
preferable  to  end  a  battle  vith  a  higher  combat  ratio  than  existed  prior  to 
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Peaks 


Controller:  Bde  S3 
Event:  Request  SITREPS. 


Insertion  Time 

Desired:  1  1/2  hours  Into  battle 
Actual: 


Bxpected, Response  Actual  Response 

Bn  S3  should  exanine  records 
and  provide  Bde  vith  situation 
report. 


From  whom:  Bn  S3 


To  vhoa:  Bde  S3 


Timeliness:  Immediate 


Completeness: 

1.  Front  line  trace. 

2.  Friendly  losses. 

3.  Probability  of  completing 
mission. 


Accuracy: 

Compare  vith  computer  results 
IS  min  prior. 


Figure  5.  Example  of  a  probe  inserted  by  the  brigade  S3  controller  to  measure 
the  performance  of  the  battalion  S3 
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Table  2 


Definitions  of  Simulation  Outcome  Measures 


Measure 


Definition 


RER  -  OPFOR  Losses/OPFOR  Initial  Strengths 

Friendly  Losses/Friendly  Initial  Strengths 


SMFRD  - 

C2ILL  - 


msn&L  Siansih  _  OPPOR  Remaining  strength 

Friendly  Initial  Strength  OPFOR  Initial  Strength 


1  /Friendly.  Remaining  Strength^  +  OPFOR  Remaining  Strength 

2  y  Friendly  Initial  Strength  J  OPFOR  Initial  Strength 


-Q.PgQR  InUlftmifflttlfr _  _  OPFOR  Remaining  Strength 

CR  -  Frj^lY  Initial  SUWMtth _ Friendly  Remaining  Strength 

OPFOR  Initial  Strength 
Friendly  Initial  Strength 


battle.  The  measure,  therefore,  computes  the  change  in  combat  ratio  relative 
to  Initial  combat  ratio. 

In  all  the  above  measures,  higher  values  Indicate  greater  success  for  the 
friendly  forces.  Initial  strength  and  losses  were  based  on  equipment  and  not 
personnel.  All  types  of  combat  equipment  were  considered  in  the  calculations, 
vhere  tanks,  APCs,  TOVs,  etc.,  vere  combined.  Combination  vas  achieved  by 
adding  the  products  of  each  equipment  type  and  its  corresponding  combat  effec¬ 
tiveness  weight.  The  CATTS  battle  calculus  included  combat  effectiveness 
weights  for  eacr.  piece  cf  equipment  based  or.  its  ability  to  destroy  other 
types  of  equipment,  e.g.,  an  M60A1  tank  had  a  weight  of  73,  an  M113  APC  a 
weight  of  *9,  and  a  T62  a  weight  of  80.  All  tour  outcome  measures  were  sig¬ 
nificantly  correlated  with  controller  ratings  of  overall  command  group  effec¬ 
tiveness  in  CATTS  exercises,  provided  that  system  and  scenario  characteristics 
(such  as  combat  ratio,  mission,  and  weather)  vere  held  constant. 

Recommendations .  A  variety  of  measures  should  bo  used  to  assess  differ¬ 
ent  aspects  of  command  group  performance.  Performance  ratings  are  easy  to 
obtain  from  exercise  controllers  and  from  the  players  themselves,  once  suit¬ 
able  rating  scales  hove  been  developed.  But  their  usefulness  is  limited  by 
the  rater’s  ability  to  discriminate  among  different  levels  of  performance  and 
between  different  tasks,  as  well  as  by  Individual  differences  among  raters. 
More  objective  measures  have  been  developed  to  overcome  these  limitations, 
Including  measures  of  information  flow,  responses  to  critical  events  (probes), 
and  simulation  outcome  measures.  These  measures  are  useful  for  providing 
players  with  feedback  to  help  them  improve  their  performance  and  for  providing 
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th«  trainers  with  feedback  to  help  improve  the  training  system.  Further 
research  should  be  done  to  develop  objective  measures  of  planning  and  deci¬ 
sionmaking  behavior,  to  develop  automated  measures  of  performance,  and  to  make 
use  of  advances  in  audio-  and  video-recording  technology. 


TRAINING  EFFECTIVENESS 

Battle  simulations  have  both  virtues  and  deficiencies.  Research  with 
CAMMS  and  ARTBASS  Identified  a  number  of  specific  strengths  and  weaknesses, 
which  are  described  in  Lessons  15-17. 

LESSON  15_ -  Command  group  members  perceived  automated  battle  simulations 
as  more  realistic  and  more  instructive  than  conventional,  manual  command  post 
exercises  (CPXsl.  In  one  investigation,  questionnaires  were  administered  to 
50  battalion  command  groups  after  they  participated  in  CAMMS  exercises  (Kaplan 
&  Barber,  1979).  The  questionnaires  asked  the  players  to  rate  CAMMS  and  a 
manual  CPX  on  several  measures  of  training  effectiveness.  CAMMS  was  judged 
significantly  and  consistently  more  realistic  than  a  CPX.  The  greatest  advan¬ 
tages  of  CAMMS  vere  exposure  to  the  capabilities  of  enemy  weapons  systems. 
Easing  1  tbijKlQg,.e»19"iY»  and  making  decisions  under  real-time  constraints. 

The  tasks  that  CAMMS  exercised  best  were  Develop  plan  based  on  mission.  Pre¬ 
pare  and_ organise  the  battlefield.  Control  and  coordinate  combat  operations. 
Employ  fires  and  other  combat  support  assets,  and  Concentrate/shift  combat 
Dover. 

Similar  results  were  obtained  for  ARTBASS  (Kaplan,  1984),  when  members  of 
22  battalion  command  groups  Indicated  how  much  they  learned  about  their  train¬ 
ing  objectives.  The  highest  rated  tasks  were  Prepare  and  issue  orders.  Pre¬ 
pare  plan/order.  Monitor  operations.  Control  and  coordinate  Bn/TF  operations. 
Plan  fire  support,  and  Control  fire  support.  Table  3  contains  a  more  complete 
list  of  the  tasks  that  ARTBASS  trained  best,  grouped  by  position.  These  tasks 
were  selected  as  training  objectives  by  a  majority  of  the  players  in  the  spec¬ 
ified  position  and  received  high  ratings  (>  3.5  on  a  five-point  scale)  on 
amount  learned.  ARTBASS  was  an  effective  command  and  control  trainer  for  the 
battalion  commander,  S2,  S3,  and  FSO.  It  trained  about  half  the  S4’s  tasks 
vtry  well,  but  none  of  the  objectives  selected  by  a  majority  of  the  Si’s. 

LESSON  16  -  Battle  simulations  trained  some  command  group  members  better 
than  others.  Table  4  compares  the  results  obtained  with  two  different  mea¬ 
sures  of  training  effectiveness.  The  command  group  members  in  CAMMS  rated  the 
similarity  of  the  tasks  they  performed  in  the  exercise  to  their  actual  job 
requirements.  The  players  in  ARTBASS  rated  how  much  they  learned  about  com¬ 
mand  group  processes,  e.g.,  acquisition  of  information,  communication,  deci¬ 
sion  making,  and  execution.  Both  aspects  of  training  effectiveness  were  rated 
on  five-point  scales  with  five  at  the  high  end  of  the  scale.  In  spite  of  the 
differences  in  method  and  in  the  simulations,  the  average  ratings  for  each 
position  were  similar  in  both  investigations.  In  both  CAMMS  and  ARTBASS  the 
S3,  battalion  commander,  and  S2  received  the  most  training,  followed  by  the 
FSO  and  S4,  while  the  SI  was  exercised  least. 


13 


Table  3 


Objective*  thrt  AKTBASS  Trained  Beat  (Mean  Matins  on  a  Five-Point  Scale) 


Petition  Objective  Rating 


Battalion  Cownosr 

Plan  fire  support  4.3 

Control  and  coordinate  Bn /TP  operations  4.2 

Control  fire  support  4.0 

Formulate  tentative  plan  3.9 

Plan  direct  lire*  3.8 

Conduct  preliminary  analysis  3.7 

Issue  OPORO  3.7 

Conduct  commander  and  staff  planning  procedures  3.6 

Prepare  plan/order  3.6 

Plan  maneuver  control  measures  3.6 

Bvaluate  mission  3.6 

52  Coordinate  with  the  BN/TP  HQs  4.0 

Obtain  information  and  Intelligence  3.8 

Prepare  plan/order  3.8 

Prepare  Intelligence  estimates  3.7 

Prepare  analysis  of  area  of  operation  3.7 

Identify  intelligence  requirements  3.7 

53  Prepare  and  issue  orders  4.3 

Prepare  plan/order  4.4 

Control  and  coordinate  Bn /TP  operations  4.3 

Monitor  operations  4.3 

Develop  task  organisation/concept  of  operation  4.2 

Coordinate  engineer  support  3.9 

Prepare  operations  estimate  3.9 

Plan  maneuver  control  measures  3.9 

Complete  mission  analysis  3.8 

Plan  fire  support  3.8 

Formulate  tentative  plans  3.8 

Conduct  preliminary  analysis  3.8 

Develop  fire  support  plans  3.8 

Control  unit  movement  and  fires  using  graphic  control  measures  3.8 

54  Plan  and  coordinate  logistical  support  3.8 

Prepare  logistical  estimates  3.7 

FSO  Control  fire  support  4.3 

Develop  fire  support  plans  4.1 

Plan  fire  support  3.7 
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Table  4 


Simulation  Training  by  Position  (Mean  Rating  on  a  Five-Point  Scale) 


Position 

CAMMS 

(Similarity) 

ARTBASS 

(Amount  Learned) 

S3 

4.3 

4.3 

Battalion  Commander 

4.0 

3.8 

S2 

4.0 

3.8 

FSO 

3.6 

3.7 

S4 

3.6 

3.3 

SI 

3.3 

2.8 

LESSON  17  -  Training  simulations  had  certain  deficiencies.  The  principal 
veaknesses  of  CAMMS  (Kaplan  4  Barber,  197S )  vere  that  it  did  not  produce  much 
stress  and  did  not  exercise  the  players  in  security  procedures,  such  as  elec¬ 
tromagnetic  and  communications  security.  Nor  did  CAMMS  require  the  players  to 
react  to  special  situations,  such  as  enemy  jamming  or  chemical,  biological,  or 
nuclear  warfare.  The  conventional  CPX  also  received  lov  ratings  in  these 
areas. 

In  field  tests  of  the  pre-production  model  of  ARTBASS  (Kaplan,  1984)  the 
players  recommended  a  number  of  improvements  in  the  system.  The  following 
recommendations  were  among  the  most  frequent: 

1.  Provide  a  separate  facility  where  the  battalion  commander  or  the 
commander  and  S3  can  see  the  battle  on  a  visual  display. 

2.  Provide  more  feedback  about  command  group  performance  in  the  TOC. 

3.  Increase  participation  by  brigade  staff  or  brigade  level  controllers. 

4.  Add  NBC  and  EV. 

5.  Improve  the  realism  of  simulated  operations.  Program  air-to-air 
engagements,  helicopters,  thermal  sites. 

6.  increase  activity  in  personnel  administration.  Add  company  XOs,  lSGs 
to  player  staff.  Incorporate  medical  play. 

7.  Improve  simulation  of  intelligence  play.  Increase  intelligence 
buildup. 

8.  Increase  activity  in  logistics.  Add  more  detail.  Require  resupply. 

9.  Improve  simulation  of  fire  support.  Although  the  PSOs  reported  that 
they  learned  a  lot  from  the  exercise,  they  recommended  a  variety  of 
Improvements. 
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Tabic  5  lists  the  command  group  training  objectives  that  ARTBASS  trained 
least  successfully!  i.e.,  the  tasks  selected  as  training  objectives  by  a 
majority  of  the  players  at  each  position  vhich  were  rated  belov  3.0  (some)  on 
a  five-point  scale  of  amount  learned,  lapleaentlng  the  players'  recommenda¬ 
tions  vould  correct  many  of  the  weaknesses  in  ARTBASS. 


Table  5 

Objectives  that  ARTBASS  Trained  Least  (Mean  Rating  on  a  Five-Point  Scale) 


Position 

Objective 

Rating 

SI 

Prepare  and  submit  personnel  dally  summary  (PDS) 

2.4 

Prepare  estimates 

2.4 

Process  casualty  feed  reports 

2.0 

Process  replacements 

1.4 

S4 

Supervise  logistical  operations 

2.9 

Request  supplies 

2.3 

FSO 

Coordinate  joint  air  attack  teams  (JAAT)  operations 

2.7 

Recommendations.  Battle  simulations  generally  train  commanders,  opera¬ 
tions,  fire  support,  and  intelligence  better  than  logistics  and  administra¬ 
tion.  Besides  making  simulations  more  complete,  it  vould  also  be  desirable  to 
develop  modular  systems  that  train  individual  staff  sections.  In  addition  to 
providing  each  staff  element  vith  optimally  focused  training,  such  part-staff 
trainers  could  be  used  providing  supplemental  training  to  less  experienced 
staff  members.  At  corps  and  division  levels  part-staff  trainers  vould  be  an 
economical  alternative  to  large-scale  exercises  for  the  entire  staff. 


FEEDBACK 

The  term  feedback  refers  here  to  Information  about  their  performance  that 
the  command  group  members  receive  during  a  training  exercise.  Lessons  18-20 
discuss  the  requirement  for  feedback  in  battle  simulations. 

LESSON  18  -  Training  simulations  provided  the  players  vith  incomplete 
feedback  about  their  performance.  The  command  group  members  were  generally 
unable  to  trace  the  remote  or  delayed  consequences  of  their  actions.  This 
limitation  applied  particularly  to  the  training  objectives  that  were  exercised 
during  the  planning  phase  before  the  simulated  battle  began.  Nor  did  the 
players  realize  vhen  they  failed  to  transmit  or  receive  essential  items  of 
Information.  They  could  not  see  the!”  own  performance  from  the  perspective  of 
the  brigade,  company,  or  threat  controllers.  Unlike  the  controllers  vho  had 
observed  many  previous  groups,  the  players  could  not  compare  their  performance 
vith  that  of  other  groups.  Thus,  merely  participating  in  the  simulation  did 
not  by  itself  diagnose  deficiencies  or  suggest  improvements. 
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LESSON  19  -  Without  added  feedback,  simulation  training  produced  limited 
improvement  In  command  group  performance.  Five  battalion  command  groups  par¬ 
ticipated  in  a  4-day  CATTS  exercise  with  only  intrinsic  feedback,  i.e.,  only 
the  knowledge  of  results  available  through  their  own  observations  (Thomas, 
Barber,  &  Kaplan,  1984).  They  received  a  complete  after  action  review  (AAR), 
but  only  at  the  end  of  the  last  exercise  day.  The  command  group  members  and 
the  brigade  controllers  reported  a  significant  improvement  in  ARTEP  task  per¬ 
formance  from  the  first  to  the  fourth  day  or  the  exercise.  However,  there 
were  no  significant  improvements  in  the  player-controllers’  (company  command¬ 
ers’)  ratings  of  group  performance  in  simulation  outcome  measures  or  in  intra- 
and  inter-group  communication,  as  measured  by  the  information-flow 
methodology. 

LESSON  20  -  Providing  additional  feedback  during  an  exercise  increased 
the  improvement  in  performance  produced  by  simulation  training.  Five  battal¬ 
ion  command  groups  were  given  detailed  diagnostic  feedback  about  their  perfor¬ 
mance  on  the  second  day  of  a  4-day  CATTS  exercise  (Thomas,  Kaplan,  &  Barber, 
1984).  On  the  first  exercise  day,  performance  was  measured  by  ARTEP  task 
ratings,  information-flow  scores,  probes,  and  simulation  battlefield  outcomes. 
The  command  group  members  received  feedback  about  all  these  measures  on  the 
morning  of  the  second  day.  They  continued  training  on  the  second  and  third 
days  and  were  evaluated  again  on  the  fourth  day.  Every  performance  measure 
increased  significantly  from  the  first  to  the  fourth  exercise  day  as  shown  in 
Table  6,  which  compares  the  improvement  with  diagnostic  feedback  to  that 
obtained  in  the  previous  study  without  added  feedback  during  the  exercise. 


Table  6 

Improvement  in  Command  Group  Performance  With  and  Without  Feedback 
(Statis' ical  Significance) 


Performance  Measure 


With  Feedback  W  thout  Feedback 


ARTEP  Ratings 

Players 

P 

<  .05 

P  < 

.01 

Controllers 

P 

<  .0001 

P  < 

.001 

Player-cont rollers 

P 

<  .01 

Not 

significant 

Information  Flow 

Brigade  to  Battalion 

P 

<  .05 

Not 

significant 

Within  the  Bn  Cmd  Grp 

P 

<  .01 

Not 

significant 

Battalion  to  CO  Cdrs 

P 

<  .01 

Not 

significant 

Probes 

P 

<  .05 

Not 

used 

Simulated  Battle  Outcomes 

P 

<  .05 

Not 

significant 
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Recommendations .  Merely  participating  In  an  exercise  does  not  necessar¬ 
ily  improve  command  group  performance.  To  maximize  the  training  benefit  of  an 
exercise,  the  group  members  should  be  given  extensive  feedback  about  their 
performance.  In  particular,  they  should  be  given  feedback  relevant  to  their 
training  objectives.  The  optimum  content,  duration,  and  format  of  feedback 
remain  to  be  determined,  but  it  should  incorporate  the  playback  capability  of 
automated  simulations,  audio-video  inputs,  and  objective  performance  measures, 
as  they  continue  to  be  developed. 


SUMMARY 

The  process  of  training  system  development  begins  with  the  determination 
of  objectives,  which  establish  a  basis  for  designing  the  system  and  for  envel¬ 
oping  performance  measures.  The  measurement  data  obtained  during  training 
exercises  are  used  to  provide  the  system  designers  and  operators  with  feedback 
about  the  effectiveness  of  their  system.  Lessons  learned  in  research  on  com¬ 
mand  group  training  were  grouped  into  four  categories  derived  from  this  model: 
(1)  training  objectives,  (2)  performance  measures,  (3)  training  effectiveness, 
and  (4)  feedback. 

The  objectives  of  a  command  group  training  system  were  defined  as  those 
ARTEP  tasks  exercised  by  the  system  that  were  both  highly  correlated  with 
overall  performance  ratings  and  frequently  deficient.  Training  objectives 
were  found  to  be  mission-dependent  and  to  vary  from  one  command  group  to 
another. 

Performance  measures,  derived  from  the  training  objectives,  included 
ratings  of  task  performance  by  the  exercise  controllers.  The  rating  scales 
had  to  be  carefully  defined,  but  still  did  not  discriminate  well  among  differ¬ 
ent  tasks  and  were  not  very  reliable.  To  remedy  these  deficiencies,  more 
objective  measures  of  information  flow,  responses  to  critical  incidents 
(probes),  and  simulation  outcome  measures  were  developed. 

Automated  battle  simulations  were  generally  perceived  as  realistic  by  the 
command  groups  that  they  trained,  although  they  trained  operations  and 
intelligence  functions  better  than  logistics  and  administration.  Merely 
participating  in  an  exercise  did  not  necessarily  improve  a  command  group's 
performance.  However,  providing  additional  feedback  about  task  performance 
ratings,  information  flow,  critical  incidents,  and  simulation  outcomes  helped 
the  group  members  to  increase  all  those  measures  of  their  performance. 

The  methods  developed  in  this  research  can  be  used  by  system  developers 
and  operators  to  refine  training  objectives,  to  assess  command  group  perfor¬ 
mance  and  training  system  effectiveness,  and  to  provide  feedback  about  per¬ 
formance  that  enhances  training  effectiveness.  The  results  obtained  with 
these  methods  suggest  several  promising  lines  of  future  research.  In  the  area 
of  performance  measurement,  further  research  should  be  done  to  develop  mea¬ 
sures  of  planning  and  decision-making  behavior,  to  automate  performance 
measurement,  and  to  utilize  advances  in  audio-  and  video-recording  technology. 
Training  effectiveness  can  be  increased  by  developing  modular  simulations  to 
train  individual  staff  sections.  Such  part-staff  trainers  would  be  especially 
economical  at  corps  and  division  level.  Finally,  training  effectiveness  can 
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be  Increased  by  further  research  on  the  type  and  amount  of  feedback  provided 
to  players  during  the  exercise. 


* 
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APPENDIX  A 


TRAINING  OBJECTIVES 
SELECTED  IN  ARTBASS  EXERCISES 


Objectives 


Percent  Selected 


Battalion  Commanders 

Conduct  commander  and  staff  planning  procedures  92 
Control  and  coordinate  Bn/TF  operations  92 
Prepare  plan/order  83 
Issue  OPORD  83 
Plan  fire  support  75 
Issue  planning  guidance  67 
Plan  maneuver  control  measures  67 
Evaluate  mission  58 
Formulate  tentative  plan  5P 
Issue  varning  order  1C 
Control  fire  support  58 
Conduct  preliminary  analysis  50 
Prepare  estimates  50 
Plan  direct  fires  50 
Control  unit  movement  and  fires  using  graphic 

control  measures  50 
Issue  FRAGO  50 
Control  direct  fires  50 
Complete  mission  analysis  42 
Maintain  orientation  33 
React  to  indirect  fires  33 

51 

Process  replacements  64 
Prepare  and  submit  personnel  dally  summary  (PDS)  64 
Prepare  estimates  55 
Process  casualty  feed  reports  55 
Forecast  losses  (CMBT)  45 
Coordinate  medical  support  45 
Prepare  plan/order  27 

52 

Obtain  information  and  intelligence  92 
Prepare  intelligence  estimates  83 
Prepare  analysis  of  area  of  operation  >  75 
Identify  intelligence  requirements  75 
Prepare  plan/oru^r  67 
Coordinate  vithin  Bn/TF  HQs  50 
React  to  direct  fire  42 


A- 1 


Objectives 


Percent  Selected 


S3 

Prepare  plan/order  92 
Complete  mission  analysis  83 
Control  and  coordinate  Bn/TF  operations  83 
Develop  task  organization/concept  of  operation  75 
Plan  fire  support  75 
Coordinate  engineer  support  75 
Prepare  and  issue  orders  75 
Formulate  tentative  plan  67 
Evaluate  mission  58 
Conduct  preliminary  analysis  58 
Develop  fire  support  plans  58 
Monitor  operations  58 
Control  unit  movement  and  fires  using  graphic 

control  measures  58 
Issue  FRAGO  58 
Prepare  operations  estimate  50 
Plan  maneuver  control  measures  50 
Issue  warning  order  42 
Control  fire  support  42 
Plan  direct  fires  33 
Coordinate  joint  air  attack  team  (JAAT)  operations  33 
Coordinate  air  defense  support  17 
Maintain  orientation  17 
Control  direct  fires  17 
React  to  indirect  fires  17 


S4 

Plan  and  coordinate  logistical  support  92 
Prepare  plan/order  75 
Prepare  logistical  estimates  67 
Supervise  logistical  operations  67 
Request  supplies  58 
Receive  supplies  33 

FSO 

Develop  fire  support  plans  92 
Plan  fire  support  67 
Control  fire  support  67 
Coordinate  joint  air  attack  team  (JAAT)  operations  58 
React  to  indirect  fire  42 


APPENDIX  B 


INFORMATION  FLOW  QUESTIONNAIRE 

Position  _  Date _ 

Answer  all  questions.  Circle  the  letter  before  the  answer  that  you 
believe  is  correct.  Do  not  guess.  If  you  do  not  know  the  answer,  answer 
"unknown."  Some  questions  may  cover  information  to  which  you  did  not  have 
access. 

1.  Your  TF  has  been  instructed  to  go  to  BP _ after  passage  of  lines. 

a.  A 

b.  D 

c.  E 

d .  unknown 

2.  FM  radio  listening  silence  will  be  imposed  until  _ . 

a.  lifted  by  TF  headquarters 

b.  reaching  the  SP 

c.  reaching  the  RP 

d .  unknown 

3.  The  minefields  and  obstacles  emplaced  in  your  sector  by  the  201st  ACR 
_  by  the  engineers. 

a.  are  planned 

b.  are  in  the  process  of  being  verified 

c.  have  been  verified 

d .  unknown 

4.  Which  one  of  the  following  statements  is  correct? 

a.  U.S.  forces  may  initiate  fire  with  the  TF  commander’s  approval. 

b.  The  covering  force  may  return  fire  if  fired  upon. 

c.  U.S.  forces  may  fire  at  any  movement  on  the  0PF0R  side. 

d .  unknown 

5.  The  ADA  weapons  control  status  is  _ . 

a.  free 

b.  tight 

c.  hold 

d .  unknown 

6.  Attack  helicopters  available  to  your  TF  for  planning  purpose  are 


a.  one  platoon 

b.  two  platoons 

c.  one  troop 

d .  unknown 


B-l 
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aircraft  are  available  for  close  air  support. 


a.  F-lll 

b.  A- 10 

c.  F-15 

d .  unknown 

8.  Close  air  support  response  time  for  strip  alert  is  _  minutes. 

a.  10  to  20 

b.  30  to  40 

c.  50  to  60 

d .  unknown 

9.  Medical  evacuation  will  be  conducted  to  the  _ . 

a.  52nd  Mech  Div  TNS 

b.  1-23  Cav  Field  TNS 

c.  TF  IKE  TNS 

d .  unknown 

10.  Personnel  replacements  are  available  within  _ . 

a.  12  hours 

b .  one  day 

c .  two  days 

d.  unknown 

11.  Wounded  POWs  are  evacuated  through  medical  channels  _ . 

a.  with  US  wounded 

b.  after  US  wounded 

c.  as  directed 

d .  unknown 

12.  Local  private  dwellings  and  business  are  _ . 

a.  off-limits  to  all  soldiers 

b.  off-limits  except  vith  owner’s  permission 

c.  available  for  TF  use 

d .  unknown 

0PF0R  will  have  _  during  the  first  12  hours  of  hostilities. 

a.  air  superiority 

b.  air  purity 

c.  no  air  support 

d .  unknown 

14.  0PF0R  aerial  reconnaissance  has  _  significantly  in  the  past  48 

hours. 


a. 

b. 


increased 

decreased 
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c.  not  changed 

d .  unknown 

15.  The  covering  force  can  expect  to  be  opposed  by  the  _ . 

a.  49th  Motorized  Rifle  Division 

b.  17th  Motorized  Rifle  Division 

c.  23rd  Motorized  Rifle  Division 

d .  unknown 

16.  OPFOR  has  the  capability  to  employ  nuclear  as  well  as  chemical  and 

biological  weapons.  Current  indications  are  that  _ . 

a.  chemical  and  nuclear  weapons  probably  will  be  used 

b.  nuclear  weapons  units  have  been  deployed  well  forward 

c.  OPFOR  has  given  no  indication  they  vill  use  NBC  veapons 

d .  unknown 

17.  OPFOR  units  are  estimated  to  be  at  _  strength. 

a.  97% 

b.  100% 

c.  103% 

d .  unknown 

18.  There  is  _  guerilla  activity  in  the  brigade  area. 

a.  increased 

b.  little 

c.  no  noticeable 

d .  unknown 

19.  Attachments  should  come  _ . 

a.  with  their  CSS 

b.  without  CSS 

c.  with  CSS  as  needed 

d .  unknown 

20.  Requests  for  replacements  of  major  end  items  vill  require  at  least 
_  hours  to  fill  after  hostilities  begin. 

a.  12 

b.  48 

c.  24 

d .  unknown 

21.  The  CSR  for  TOVs  is  _  rounds. 

a.  3 

b.  6 

c.  9 

d .  unknown 
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22.  Aerial  resupply  is  _ . 

a.  available  from  corps 

b.  limited 

c.  available  from  division 

d.  unknown 

23.  Convoy  movements  may  be  conducted  only 


a. 

during  darkness 

b. 

from  dusk 

to 

dawn 

c. 

from  2400 

to 

0400 

d. 

unknovn 

24.  There  are  shortages  of  _  rounds. 

a.  HE 

b.  smoke 

c.  DPICM 

d.  unknown 

25.  No  more  than  _  of  CSR  will  be  expended  to  support  covering  force 

operations. 


a. 

40X 

b. 

50* 

c. 

60X 

d. 

unknovn 

26.  Requests  for  quick  smoke  must  be  approved  by  the  _ . 

a.  FIST  chief 

b.  company  commander 

c.  TF  commander 

d.  unknown 

27.  Aerial  observation  is  _ . 

a.  not  available  due  to  shortages  of  aircraft 

b.  available  150800 

c.  available  for  high  priority  missions 

d .  unknovn 

28.  Artillery  should  be  used  on  targets  of  _  or  more  vehicles. 

a.  3 

b.  5 

c.  7 

d .  unknovn 
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APPENDIX  C 


EXAMPLES  OP  PROBP1* 
Probe 


Controller:  Co  XO 
Event:  Coapeny  Inputs 

lncoaplete  strength 
reports. 

Expected  Response 

Bn  Si  should  recognise 
incompleteness  and  request 
additional  information. 


Insertion  Time 

Desired:  After  losses  have  occurred 
Actual: 


Actual  Response 


From  vhom:  Bn  SI 


To  vhom:  Co  XO,  Pit  Ldrs, 
or  Pit  Sgts 


Timeliness:  Immediate 


Completeness: 

Request  all 

incomplete 

lines. 


Accuracy: 

Request  appropriate 
items  only. 


C-l 


Prob« 


Controller!  Bde  S2 
Event!  Request  froe  Bde  S2 

for  estleate  of  OPPOR 
unit  TP  is  feeing 

Banc  ttd  Bmami 

Bn  S2  detereines  unit 
type  through  intelligence 
gathered  by  S2  sources. 


InaifUon  Ilia 

Desired!  1  1/2  hours  after  initial 
contact 

Actual! 

Actual  Response _ 


From  vhoat  Bn  S2 


To  vhoai  Bde  S2 


Tiaeliness!  Iauned late  upon 
request 


1 


Completeness: 

1.  Unit  sise 

2.  Unit  type 

3.  Main  attack? 


Accuracy: 

Compare  S2  estimates 
vith  actual  OPPOR. 
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Probe 


Controller:  Bde  S4 
Bvent:  Final  estimates  of 
equipeent  strength 


Insertion  Time 

Desired:  IS  Minutes  before  end  of 
exercise 

Actual: 


Expected  Response  AsiM&l  Bsaansa 

Bn  S4  revievs  records  and 
reports  tc  Bde. 


From  vhom:  Bn  S4 


To  vhom:  Bde  S4 


Timeliness:  Within  IS  minutes 


Completeness: 

All  equipment  Included 


Accuracy: 

Compare  vith  actual  losses 
15  minutes  prior. 
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Prate 


Con t roller t  FIST 
Event:  FIST  calls  for 
fires  out  of 
sector. 


Desired:  During  intense  Arty  opns 
Actual: 


f  VvV  M  V  l  \ 


Bn  FSO  should  identify  and 
cancel  fires. 


To  whom:  FIST  or  FDC 


Timeliness:  Immediate 


Correctly  cancels  fires 


